Image processing device and image processing method

ABSTRACT

There is provided an image processing device including a detection unit configured to detect a mask from an acquired image, a determination unit configured to determine whether there is a change in the mask detected by the detection unit, and an output unit configured to output a parameter when the determination unit determines that there is a change in the mask, the parameter being related to the mask detected by the detection unit before it is determined that there is a change in the mask.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2014-032809 filed Feb. 24, 2014, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present technology relates to an image processing device and animage processing method and, more particularly, the present technologyrelates to an image processing device and an image processing method,capable of appropriately correcting a mask of an endoscope.

An endoscope is used as a medical instrument that is inserted into thebody of a subject such as patients and observes the inside of the body.An image from an endoscope is displayed in the form of a circular frameon a rectangular screen. In such cases, it is necessary to detect animage portion in distinction from a lens barrel portion that isdisplayed as a portion shaded by a lens barrel.

A mask may be used to distinguish a portion that provides an imageobtained by an endoscope for the user from a portion that does notprovide an image for the user, and JP 2012-125469A discloses a method ofdetecting a mask of an endoscope.

A mask of an endoscope is detected and software processing is allowed tobe performed only in a mask that is an effective area, and thus it ispossible to reduce the amount of calculation and to minimize the adverseeffects associated with a peripheral portion of a mask in a filterprocessing and other like processing.

SUMMARY

The coupling between an endoscope and a camera head may be loose, andthus the endoscope is likely to be shifted or rotated during surgery. Tosolve this, it is necessary to detect a mask for every frame or at everycertain interval.

However, as disclosed in JP 2012-125469A, if it is intended to detect amask from an endoscopic image obtained by capturing the inside of thebody, erroneous detection may be caused depending on conditions of anendoscopic image, for example, inaccurate capturing of boundary betweena mask and other portions, existence of obstacles such as bubbles, andadhesion of dirt on a lens.

In this way, the position of a mask of an endoscope may be shifted, andthus it is desirable to detect and correct a mask in such a case whenthe position is shifted.

The present technology is made in view of such circumstances, and it isintended that the correction of a mask is allowed to be performed withaccuracy.

According to an embodiment of the present disclosure, there is providedan image processing device including a detection unit configured todetect a mask from an acquired image, a determination unit configured todetermine whether there is a change in the mask detected by thedetection unit, and an output unit configured to output a parameter whenthe determination unit determines that there is a change in the mask,the parameter being related to the mask detected by the detection unitbefore it is determined that there is a change in the mask.

The determination unit may determine whether there is a change in themask based on a temporal change in a parameter of the mask detected bythe detection unit.

The determination unit may determine that there is no change in the maskdetected by the detection unit when a difference between a parameter ofa first mask detected by the detection unit and a parameter of a secondmask detected by the detection unit after the first mask is detected isless than a predetermined threshold. The output unit may output aparameter of a third mask when the determination unit determines thatthere is no change in the mask, the parameter of the third mask beingcalculated from the parameter of the first mask and the parameter of thesecond mask.

The determination unit may determine that there is a change in the maskdetected by the detection unit when a difference between a parameter ofa first mask detected by the detection unit and a parameter of a secondmask detected by the detection unit after the first mask is detected isgreater than or equal to a predetermined threshold. The output unit mayoutput the parameter of the first mask when the determination unitdetermines that there is a change in the mask.

The output unit may output the parameter of the second mask instead ofthe parameter of the first mask when the determination unit determinesthat there is a change in the mask by a predetermined number of times.

The output unit may output a parameter of a third mask calculated fromthe parameter of the first mask and the parameter of the second maskwhen the determination unit determines that there is a change in themask by a predetermined number of times.

An image in the mask may include an image captured by an endoscope.

The image processing device may further includes a correction unitconfigured to correct a parameter of the mask using a reliabilityparameter representing reliability of a parameter of the mask.

The detection unit may detect a first mask and a second mask from tworespective acquired images. The image processing device may furtherinclude a correction unit configured to determine whether at least oneof a parameter of the first mask and a parameter of the second mask isto be corrected, based on the parameter of the first mask and theparameter of the second mask.

The correction unit may correct one of the parameter of the first maskand the parameter of the second mask using the other parameter that isset previously, when a difference between the parameter of the firstmask and the parameter of the second mask is outside a predeterminedrange.

The correction unit may perform correction using a reliability parameterrepresenting reliability of each of the parameter of the first mask andthe parameter of the second mask when a difference between the parameterof the first mask and the parameter of the second mask is outside apredetermined range.

The determination unit may determine whether there is a change in themask based on a luminance level of the image.

The determination unit may determine that there is a change in the maskwhen an average luminance level of the image is less than apredetermined threshold.

The determination unit may determine whether there is a change in themask based on whether an endoscope is inserted into a trocar.

The determination unit may determine that there is a change in the maskwhen it is determined that an endoscope is not inserted into a trocar.

The determination unit may determine whether there is a change in themask based on an external input.

The external input may be provided using a foot switch.

According to another embodiment of the present disclosure, there isprovided an image processing method including detecting a mask from anacquired image, determining whether there is a change in the detectedmask, and outputting a parameter when it is determined that there is achange in the mask, the parameter being related to the mask detectedbefore it is determined that there is a change in the mask.

In the image processing device and image processing method according toan embodiment of the present technology, a mask is detected from anacquired image, whether there is a change in the detected mask isdetermined, and when it is determined that there is a change in themask, a parameter related to the mask detected before it is determinedthat there is a change in the mask is outputted.

According to an embodiment of the present technology, the correction ofa mask is allowed to be performed with accuracy.

Note that the advantages herein are not necessarily intended to berestrictive, and any other advantage described in the present disclosuremay be achievable.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating the configuration of an embodiment ofan image processing device to which the present technology is applied;

FIG. 2 is a diagram illustrated to describe a mask;

FIG. 3 is a flowchart illustrated to describe the operation performed bythe image processing device;

FIG. 4 is a diagram illustrated to describe the correction of a maskshape:

FIG. 5 is a diagram illustrated to describe the correction of a maskshape;

FIG. 6 is a diagram illustrating the configuration of an embodiment ofan image processing device to which the present technology is applied;

FIG. 7 is a diagram illustrated to describe a mask;

FIG. 8 is a flowchart illustrated to describe the operation performed bythe image processing device;

FIG. 9 is a flowchart illustrated to describe the operation performed bythe image processing device;

FIG. 10 is a diagram illustrated to describe the correction of a maskshape; and

FIG. 11 is a diagram illustrated to describe a recording medium.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Embodiments for implementing the present technology (hereinafter simplyreferred to as “embodiment”) will be described. The description will bemade in the following order.

1. Configuration of image processing device according to firstembodiment

2. Operation by image processing device according to first embodiment

3. Configuration of image processing device according to secondembodiment

4. Operation by image processing device according to second embodiment

5. When operation is started

6. Recording medium

<Configuration of Image Processing Device According to First Embodiment>

An image processing device described herein is an image processingdevice for processing an image obtained from, for example, an endoscope.The present technology described herein may be applied to any device foracquiring an image and detecting a mask from the acquired image otherthan the device for processing an image obtained from an endoscope. Thefollowing description will be made by taking, as an example, an imageprocessing device for processing an image obtained from an endoscope.

FIG. 1 is a diagram illustrated to describe the configuration of animage processing device according to a first embodiment. The imageprocessing device 100 shown in FIG. 1 acquires image data from anendoscopic device (not shown) used as medical instruments, processes theacquired image, and outputs the processed image to a display unit 101(shown in FIG. 2) such as a monitor for displaying the image.

The image processing device 100 shown in FIG. 1 is configured to includean image acquisition unit 111 and a mask detection and correction unit112. The mask detection and correction unit 112 is configured to includea mask detection unit 121, a mask detection result buffer 122, and aninter-frame mask detection result comparison unit 123.

The image acquisition unit 111 of the image processing device 100acquires an image from an endoscopic device (not shown). The endoscopicdevice is configured to include an endoscope, a light source unit, animaging means, and a camera control unit. The endoscope forms an in-vivoimaging device inserted into the body cavity for capturing the inside ofthe body. The light source unit supplies illumination light to theendoscope. The imaging means of the endoscope may be a charge-coupleddevice (CCD). The camera control unit performs signal processing for theimaging means. The image acquisition unit 111 acquires image dataoutputted from the camera control unit.

The image data acquired by the image acquisition unit 111 is supplied tothe mask detection unit 121 of the mask detection and correction unit112. A detailed description of a mask is given with reference to FIG. 2.FIG. 2 illustrates an example of an image displayed on the display unit101. The central region of the screen is a circular effective region151, which presents an image captured by the endoscopic device to theuser.

As shown in FIG. 2, in an image obtained from the endoscope, there is aregion in which an in-vivo image is invisible at left, right, upper andlower parts of the image. This is because there is a region in whichlight is not transmitted to the imaging means due to the existence ofvignetting in the endoscope. The region in which an in-vivo image isinvisible is referred to as a mask region 152.

The shape of the boundary between the mask region 152 and an effectiveregion 151 in which an in-vivo image is visible is referred to as a maskshape. In the example of the screen shown in FIG. 2, a mask shape iscircular. The following description will be made on the assumption thata mask shape is a perfect circle, but the present technology isapplicable to other shapes including an ellipse. As shown in FIG. 2, thefollowing description will be made on the assumption that the center ofthe circle is set to center point P, coordinates of the center point areset to (X,Y), and the radius of the circle is set to R.

The mask detection unit 121 of the image processing device 100 detectsthe shape of the boundary between the effective region 151 and the maskregion 152, that is, in this case, parameters of the coordinates (X,Y)and radius R of the center point P of the circle.

For example, the mask detection unit 121 detects an edge on the boundarybetween a mask and in-vivo information in the image, performs the Houghtransform on the edge by setting the detected edge as a candidate point,estimates a circle (mask), and then detects a mask. For the maskdetection unit 121 to detect a mask, any detection method may beemployed.

Erroneous detection of a mask by the mask detection unit 121 may becaused by obtaining a low number of edges on the boundary between themask region 152 and an in-vivo region (effective region 151) that isinsufficient to perform detection, when contrast of a capturedendoscopic image is insufficient due to a lack of amount of light ordepending on the color of an in-vivo object to be captured.

Thus, the mask detection and correction unit 112 is configured toinclude the mask detection result buffer 122 and the inter-frame maskdetection result comparison unit 123 to correct a mask detected by themask detection unit 121.

A mask detection result outputted from the mask detection unit 121 issupplied to the mask detection result buffer 122 and the inter-framemask detection result comparison unit 123. The result outputted from themask detection unit 121 is delayed by one frame and is stored in themask detection result buffer 122.

The inter-frame mask detection result comparison unit 123 is configuredto include a determination unit 124 and an output unit 125. Theinter-frame mask detection result comparison unit 123 compares a maskdetection result of an image at the current time t and a mask detectionresult at time t−1 stored in the mask detection result buffer 122, whichwill be described later in detail. Then, the inter-frame mask detectionresult comparison unit 123 determines whether there is a change in themask depending on the absolute value of the difference between two maskdetection results, corrects the mask if necessary, and outputs the maskdetection result.

The description will be continued on the assumption that the resultoutputted from the mask detection unit 121 is delayed by one frame andstored in the mask detection result buffer 122. In addition, thedescription will be continued on the assumption that a process describedbelow is performed for every frame. However, the present technology isnot limited to the case in which a process is performed for every frame,but may be configured to perform a process for every several frames.

As described later, the inter-frame mask detection result comparisonunit 123 may be configured to perform a process if a predeterminedcondition is met and to output a mask shape.

<Operation by Image Processing Device According to First Embodiment>

Referring to the flowchart of FIG. 3, the operation by the imageprocessing device 100 shown in FIG. 1 is described. Specifically, theoperation performed by the inter-frame mask detection result comparisonunit 123 of the mask detection and correction unit 112 is mainlydescribed.

The process shown in the flowchart of FIG. 3 is started on the basis ofthe assumption that the mask detection result buffer 122 stores adetection result which is obtained from a process performed by the maskdetection unit 121 and delayed by at least one frame and stores acorrection result which is obtained from a process performed by theinter-frame mask detection result comparison unit 123 and delayed by atleast one frame.

In step S101, the inter-frame mask detection result comparison unit 123reads a mask shape from the mask detection unit 121. The information tobe read as a mask shape includes information regarding the coordinates(X_(t),Y_(t)) of the center point of the circle and informationregarding the radius R_(t) of the circle.

In the following description, information of a mask shape that isdetected at time t is represented by adding the letter t, for example,X_(t), Y_(t), and R_(t). Information of a mask shape that is detected atthe time of the frame immediately preceding the time t is represented byadding the letter t−1, for example, X_(t-1), Y_(t-1), and R_(t-1).

In step S102, the inter-frame mask detection result comparison unit 123reads a mask detection result after correction at time t−1 (the center(X′_(t-1), Y′_(t-1)) and radius R′_(t-1) of the circle) from the maskdetection result buffer 122.

The mask detection result buffer 122 stores information with regard to amask shape from the mask detection unit 121 and information with regardto a mask shape from the inter-frame mask detection result comparisonunit 123.

In the following description, information, which is supplied from themask detection unit 121 and stored in the mask detection result buffer122, is represented by adding the letter t−1, for example, the center(X_(t-1), Y_(t-1)) and radius R_(t-1) of the circle.

Information, which is supplied from the inter-frame mask detectionresult comparison unit 123 and stored in the mask detection resultbuffer 122, is represented by further adding the prime symbol, forexample, the center (X′_(t-1), Y′_(t-1)), and radius R′_(t-1) of thecircle.

In step S103, the absolute values of the differences between respectiveparameters of both, AD′_(X), AD′_(Y), and AD′_(R) are calculated basedon the following Equations (1) to (3). Here, AD′_(X) indicates theabsolute value of the difference between x-coordinate X_(t) of thecenter point of the mask detected at time t and x-coordinate X′_(t-1) ofthe center point of the mask after correction at time t−1.

Similarly, AD′_(Y) indicates the absolute value of the differencebetween y-coordinate Y_(t) of the center point of the mask detected attime t and y-coordinate Y′_(t-1) of the center point of the mask aftercorrection at time t−1. In addition, AD′_(R) indicates the absolutevalue of the difference between the radius R_(t) of the center point ofthe mask detected at time t and the radius R′_(t-1) of the center pointof the mask after correction at time t−1.

AD′x=|X _(t) −X′ _(t-1)|  (1)

AD′ _(Y) =|Y _(t) −Y′ _(t-1)|  (2)

AD′ _(R) =|R _(t) −R′ _(t-1)|  (3)

In step S104, it is determined whether the absolute value of thedifference between respective parameters is less than a predeterminedthreshold that is set for each parameter. In other words, it isdetermined whether the following Equations, that is, three inequalities(4) to (6) are established. In the three inequalities (4) to (6), thethreshold TH_AD′x is a threshold for the absolute value of x-coordinateof the center point, the threshold TH_AD′y is a threshold for theabsolute value of y-coordinate of the center point, and the thresholdTH_AD′_(R) is a threshold for the absolute value of the radius R.

Absolute value of x-coordinate of center point,AD′ _(X)<Threshold,TH_(—) AD′ _(X)  (4)

Absolute value of y-coordinate of center point,AD′ _(Y)<Threshold,TH_(—) AD′ _(Y)  (5)

Absolute value of radius R,AD′ _(R)<Threshold,TH _(—) AD′ _(R)  (6)

If it is determined that the three inequalities (4) to (6) are allestablished in step S104, then the process proceeds to step S105. Instep S105, a count value is set to zero. The count value is incrementedwhen any one of the three inequalities (4) to (6) is not established,and is incremented when the mask shape is likely to be changed.

Any one of the three inequalities (4) to (6) is not established whenthere is a change in the detected mask shape. As described above,erroneous detection of a mask by the mask detection unit 121 may becaused by obtaining a low number of edges on the boundary between themask region 152 and an in-vivo region (effective region 151) that isinsufficient to perform detection, when contrast of a capturedendoscopic image is insufficient due to a lack of amount of light ordepending on the color of an in-vivo object to be captured.

If a mask shape is changed for some reasons, parameters of thecoordinates or radius of the center point of the mask shape outputted asa mask shape at time t−1 have different values from parameters obtainedat time t. Thus, the difference between a parameter of the correctedmask shape at time t−1 and a parameter of the mask shape detected attime t increases as the change in a mask shape increases.

For this reason, it can be determined that there is no change in a maskshape when the three inequalities (4) to (6) are all established(condition in which the amount of change is small even if there is achange). It can be determined that there is a change in a mask shapewhen any one of the three inequalities (4) to (6) is not established.

In step S105, the count value (Cnt), which is set to zero, representsthe number of times the mask shape is determined to be changed. Thus, instep S105, it is determined that there is no change in a mask shape instep S104, and thus the count value is set to zero.

In step S106, a weighted average between a detection result obtained bythe mask detection unit 121 at time t and a mask detection result aftercorrection at time t−1 is calculated based on the following Equations(7) to (9), and a mask detection result after correction (centercoordinates (X′_(t),Y′_(t)) and radius R′_(t) of the circle) isdetermined.

X′ _(t) =w ₁ X _(t)+(1−w ₁)X′ _(t-1)  (7)

Y′ _(t) =w ₁ Y _(t)+(1−w ₁)Y′ _(t-1)  (8)

R′ _(t) =w ₁ R _(t)+(1−w ₁)R′ _(t-1)  (9)

In Equations (7) to (9), w1 represents a weighting factor that satisfies0≦w1≦1. If the weighting factor w1 is larger, the allowable range ofvariations in mask detection results between frames can be set tolarger. If the weighting factor w1 is smaller, the allowable range ofvariations in mask detection results between frames can be set tosmaller.

In step S107, the center coordinates (X′_(t),Y′_(t)) and radius R′_(t)of the circle that are calculated by inter-frame mask detection resultcomparison unit 123 in step S106 are outputted to the subsequentprocessing unit (not shown).

In this case, the three inequalities (4) to (6) are all established andat that time it is determined that the mask shape being outputted andmask shape being detected have no significant change, and thus the maskshape detected by the mask detection unit 121 may be outputted to thesubsequent processing unit (not shown) without any modification.

In the present embodiment, further in step S106, a correction process toabsorb the small amount of change in the mask shape is performed byEquations (7) to (9) and the mask shape obtained by performing suchcorrection process is outputted to the subsequent processing unit. Thus,it is possible to obtain a more accurate mask shape.

On the other hand, if it is determined that any one of the threeinequalities (4) to (6) is not established in step S104, the processproceeds to step S108. The process proceeds to step S108 when there is apossibility of a change in a mask shape.

In step S108, the inter-frame mask detection result comparison unit 123reads the mask detection result before correction at time t−1 (thecenter coordinates (X_(t-1),Y_(t-1)) and radius R_(t-1) of the circle),which is stored in the mask detection result buffer 122.

In step S109, the absolute values of the difference between therespective parameters of the detection result of the mask detection unit121 at time t (center coordinates (X_(t),Y_(t)) and radius R_(t) of thecircle) and the mask detection result before correction at time t−1(center coordinates (X_(t-1),Y_(t-1)) and radius R_(t-1) of the circle),AD_(X), AD_(Y), and AD_(R), are calculated based on the followingEquations (10) to (12).

AD _(X) =|X _(t) −X _(t-1)|  (10)

AD _(Y) =|Y _(t) −Y _(t-1)|  (11)

AD _(R) =|R _(t) −R _(t-1)|  (12)

In Equations (10) to (12), AD_(X) represents the absolute value of thedifference between respective x-coordinates of the center points of themasks detected at time t and time t−1, AD_(Y) represents the absolutevalue of the difference between respective y-coordinates of the centerpoints of the masks detected at time t and time t−1, and AD_(R)represents the absolute value of the difference between respective radiiR of the masks detected at time t and time t−1.

In step S110, it is determined whether the absolute values of thedifference between the respective parameters are all less than apredetermined threshold. In other words, it is determined whether thefollowing Equations, that is, three inequalities (13) to (15) are allestablished.

In the following three inequalities (13) to (15), a threshold TH_AD_(X)is a threshold for the absolute value of x-coordinate of the centerpoint, a threshold TH_AD_(Y) is a threshold for the absolute value ofy-coordinate of the center point, and a threshold TH_AD_(R) is athreshold for the absolute value of the radius R.

Absolute value of x-coordinate of center point,AD _(X)<Threshold,TH _(—)AD _(X)  (13)

Absolute value of y-coordinate of center point,AD _(Y)<Threshold,TH _(—)AD _(Y)  (14)

Absolute value of radius R,AD _(R)<Threshold,TH _(—) AD _(R)  (15)

In step S110, if it is determined that the three inequalities (13) to(15) are all established, the process proceeds to step S111. In stepS111, the count value is set to a value incremented by one.

On the other hand, in step S110, if it is determined that any one of thethree inequalities (13) to (15) is not established, the process proceedsto step S112. In step S112, the count value is set to zero(initialization).

In step S111 or S112, after the count value is set, the process proceedsto step S113. In step S113, it is determined whether the count value Cntis less than a predetermined threshold TH_CNT.

In step S113, if it is determined that the count value Cnt is less thanthe predetermined threshold TH_CNT, then the process proceeds to stepS114. In step S114, the mask detection result after correction at timet−1 (center coordinates (X′_(t-1), Y′_(t-1)) and radius R′_(t-1) of thecircle) is substituted for the mask detection result after correction attime t (center coordinates (X′_(t),Y′_(t)) and radius R′_(t) of thecircle).

In this case, the parameters of the mask shape after correctionoutputted to the subsequent stage at time t−1 is set as parameters ofthe mask shape after correction to be outputted to the subsequent stageat time t. The parameters that are set in this way are outputted to thesubsequent processing unit (step S107) and stored in the mask detectionresult buffer 122 as parameters of the mask shape after correctionoutputted to the subsequent stage at time t−1.

In this way, the process proceeds to step S114 when, even though thereis a possibility of a change in the mask shape, the mask is continued tokeep its shape.

On the other hand, in step S113, if it is not determined that the countvalue Cnt is less than the threshold TH_CNT, the process proceeds tostep S115. In step S115, a weighted average between the detection resultof the mask detection unit 121 at time t and the mask detection resultafter correction at time t−1 is calculated, and the mask detectionresult after correction at time t (center coordinates (X′_(t),Y′_(t))and radius R′_(t) of the circle) is determined.

This calculation is performed based on the following Equations (16) to(18). In Equations (16) to (18), w2 is a weighting factor that satisfies0≦w2≦1.

X′ _(t) =w ₂ X _(t)+(1−w ₂)X′ _(t-1)  (16)

Y′ _(t) =w ₂ Y _(t)+(1−w ₂)Y′ _(t-1)  (17)

R′ _(t) =w ₂ R _(t)+(1−w ₂)R′ _(t-1)  (18)

The case in which the process proceeds to step S115 corresponds to whenit is determined that the changed mask shape is a correct mask shapebecause the mask shape is continued to be determined to be changedduring a predetermined period of time. If the mask shape is immediatelymodified into the changed mask shape, the mask shape is likely to beabruptly changed, and thus the calculation in step S115 is performed forgradual modification into the changed mask shape.

In step S116, the count value (Cnt) is initialized to zero. Then, instep S107, the center coordinates (X′_(t),Y′_(t)) and radius R′_(t) ofthe circle, which are calculated by Equations (16) to (18), areoutputted to the subsequent processing unit and are stored in the maskdetection result buffer 122 as parameters of the mask shape aftercorrection which are outputted to the subsequent stage at time t−1.

In this way, when the absolute values of the difference, AD′_(X),AD′_(Y), and AD′_(R), are continuously respectively greater than orequal to the thresholds TH_AD′_(X), TH_AD′_(Y), and TH_AD′_(R), and theabsolute values of the difference, AD_(X), AD_(Y), and AD_(R), arecontinuously respectively less than the thresholds TH_AD_(X), TH_AD_(Y),and TH_AD_(R), over a predetermined period of time (the duration atwhich the count value is greater than or equal to the threshold TH_CNT),the position of a mask is likely to be physically shifted in the middleof processing.

In such cases, it is necessary to correct the position of a mask toobtain a new position of a mask over a certain period of time. In thiscase, if the weighting factor w2 increases, the time until thecorrection to obtain a new position of a mask is performed decreases,and if the weighting factor w2 decreases, the time until the correctionto obtain a new position of a mask is performed increases.

The parameters that are set in this way are outputted to the subsequentprocessing unit as the mask detection result after correction at time t(center coordinates (X′_(t),Y′_(t)) and radius R′_(t) of the circle) instep S107, as described above. The mask detection result aftercorrection at time t (center coordinates (X′_(t),Y′_(t)) and radiusR′_(t) of the circle) is also outputted to the mask detection resultbuffer 122 and stored therein for use in the next frame.

The above processing will be described in detail with reference to FIGS.4 and 5.

In FIGS. 4 and 5, the horizontal axis t represents time, and images(frames) of the figures are images obtained in the past in the order oftime toward the left side. In FIGS. 4 and 5, the mask region 152 isshown without being blackened.

A solid-line circle 171 in the figures represents the result outputtedfrom the mask detection unit 121 (see FIG. 1), and A dotted-line circle172 in the figures represents the result outputted from the inter-framemask detection result comparison unit 123 (see FIG. 1).

A cloud-shaped mark in the figures represents a frame in which the imageis in a deteriorated condition for some reasons such as adhesion of dustor dirt on the lens included in an endoscopic device and then theaccurate mask detection is not allowed to be performed. Such a situationis assumed to be occurred in the frames at time T3 and T4 in the exampleof FIG. 4.

During the period of time from time T0 to T2 in FIG. 4, the absolutevalue of the difference between the mask detection result beforecorrection (result outputted from the mask detection unit 121) and themask detection result after correction in the immediately previous frameis less than a threshold, the mask detection result after correction ofthe frame at the current time is calculated using Equations (7) to (9).

In other words, during the period of time from time T0 to T2, as shownin FIG. 4, the circle 171 representing the mask detection result beforecorrection and the circle 172 representing the mask detection resultafter correction are located at substantially the same position.

At time T3, it is assumed that an image is difficult to performdetection of a mask in the image for a reason such as adhesion of dustor dirt on the lens. In this case, the mask detection unit 121 fails toaccurately perform detection of a mask, which results in erroneous maskshape result being outputted.

However, the inter-frame mask detection result comparison unit 123compares the result outputted from the mask detection unit 121 with themask shape after correction in the immediately previous frame (frame attime T2). Then, the absolute value of the difference between both isgreater than or equal to the threshold, and the mask shape aftercorrection at time T2 is outputted from the inter-frame mask detectionresult comparison unit 123 as a result after correction at time T3.

Referring to the image at time T3 in FIG. 4, the circle 171 representingthe mask detection result before correction is shown as a small circleon the left, but the circle 172 representing the mask detection resultafter correction is the same circle 172 as the circle 172 at time T2.

Here, the description will be made on the assumption that the thresholdTH_CNT of the count value (Cnt) is set to five in the flowchart of FIG.3. At time T4, as in the case of time T3, even when the mask shaperesult obtained by the mask detection unit 121 is erroneous, the countvalue (Cnt) is less than the threshold TH_CNT(=5). Thus, the inter-framemask detection result comparison unit 123 corrects the erroneous maskshape to make the mask shape after correction at time T3.

At time T5, the image is in an improved condition, and the absolutevalue of the difference between the result outputted from the maskdetection unit 121 and the mask shape after correction in the frame atthe immediately previous time T4 is less than a threshold. Thus, themask shape after correction in the frame at the current time T5 iscalculated using Equations (7) to (9).

In this way, it is possible to prevent erroneous detection at a givenperiod of time (less than the threshold TH_CNT).

Referring now to FIG. 5, it is assumed that a case in which externalforce is applied to an endoscope, a connection portion between theendoscope and a camera head is shifted, and thus the mask is deviatedfrom its normal position, or a case in which an endoscope is turned andthus the mask is deviated from its normal position.

In such a case, the mask remains in the deviated position, and thusprevious detection of a mask only once is insufficient to detect a mask.

In FIG. 5, when the position of a mask is deviated at time T23 for thereason as described above, the position of a mask (mask shape) beforethe deviation of its position is outputted as an output result of theinter-frame mask detection result comparison unit 123 during the periodof time when the value is less than the threshold TH_CNT.

Referring to the image at time T23 in FIG. 5, the circle 171representing the mask detection result before correction is deviated tothe left side, but the circle 172 representing the mask detection resultafter correction is located at the same position as the circle 172 attime T22, and thus the mask at the position is presented to the user.

In this regard, when the threshold is set to five (TH_CNT=5), theposition of a mask before the deviation of its position is outputted asan output result of the inter-frame mask detection result comparisonunit 123 over the period of time from T23 to T26. Then, at time T27, itis not determined that the count value (Cnt) is less than the thresholdTH_CNT (=5), the mask detection result after correction is a valuecalculated by Equations (16) to (18) and a process is started to make itgradually close to the position of a mask after the deviation of itsposition.

In the example shown at time T27 in FIG. 5, the circle 172 representingthe mask detection result after correction is set at substantially thesame position as the circle 171 representing the detection result of amask before correction that is deviated to the left side, and thus themask at that position is presented to the user.

In this way, when the parameter of the mask shape is set to a valuecalculated by Equations (16) to (18) and is gradually close to theposition of a mask after the deviation of its position, it swiftlyconverges on the deviated position of a mask as the weighting factor w2in Equations (16) to (18) increases.

Furthermore, if the threshold TH_CNT is made excessively small, it ispossible to swiftly modify the position of a mask to the deviatedposition of a mask in the case as shown in FIG. 5, but it may be unableto appropriately deal with the erroneous detection in a short time asshown in FIG. 4. On the other hand, if the threshold TH_CNT is madeexcessively large, it is able to appropriately deal with the erroneousdetection in a short time, but in the case as shown in FIG. 5, it takestime to modify the position of a mask. In view of the above, thethreshold TH_CNT is set.

At time T0 or time T20, there is no mask shape after correction of theprevious frame. In such a case, the result outputted from the maskdetection unit 121 at that time may be used without any modification asthe result outputted from the inter-frame mask detection resultcomparison unit 123, or the detection of a mask is performed for animage in which a scene from which a mask is able to be easily detectedis captured previously, and the detected mask may be used as the maskshape after correction of the previous frame at time T0 or time T20.

Thus, according to the present embodiment, it is possible to accuratelycorrect a mask shape.

<Configuration of Image Processing Device According to SecondEmbodiment>

FIG. 6 is a diagram illustrating the configuration of an imageprocessing device according to a second embodiment. The followingdescription is made on the assumption that the image processing device200 according to the second embodiment is a device for acquiring animage from an endoscopic device and processing the acquired image, whichis similar to the image processing device 100 according to the firstembodiment.

The image processing device 200 shown in FIG. 6 is configured to includeimage acquisition units 211-1 and 211-2, mask detection and correctionunits 212-1 and 212-2, and an inter-stereo mask detection resultcomparison and correction unit 213.

The mask detection and correction unit 212-1 is configured to include amask detection unit 221-1, a mask detection result buffer 222-1, and aninter-frame mask detection result comparison unit 223-1. The maskdetection and correction unit 212-2 is configured to include a maskdetection unit 221-2, a mask detection result buffer 222-2, and aninter-frame mask detection result comparison unit 223-2.

The image processing device 200 acquires a left-eye image and aright-eye image from an endoscopic device, processes the acquiredimages, and outputs the processed image to a subsequent processing unit(not shown). Thus, a portion for processing the left-eye image and aportion for processing the right-eye image are provided, and the portionfor processing the left-eye image has a substantially similarconfiguration to that of the portion for processing the right-eye image.

The following description will be made on the assumption that, whenthere is no necessity for a distinction between the portion forprocessing the left-eye image and the portion for processing theright-eye image, either one of them is given as an example. In addition,for example, when there is no necessity for a distinction between theimage acquisition unit 211-1 and the image acquisition unit 211-2, theseunits will be simply referred to as image acquisition unit 211 in thefollowing description. This is similarly applied to other components.

The combination of the image acquisition unit 211 and the mask detectionand correction unit 212 has a similar configuration to that of thecombination of the image acquisition unit 111 and the mask detection andcorrection unit 112 of the image processing device 100 according to thefirst embodiment illustrated in FIG. 1, and substantially similarprocessing is performed by them. Thus, the detailed description thereofis omitted.

The image processing device 200 shown in FIG. 6 is used for a monocularstereo endoscope, and thus two endoscopic images for the left and righteyes are obtained. The following description is made on the assumptionthat the center coordinates of the mask shape 252-1 in the endoscopicimage for the left eye is set to coordinates (X0,Y0) and the radiusthereof is set to radius R0, and the center coordinates of the maskshape 252-2 in the endoscopic image for the right eye is set tocoordinates (X1,Y1) and the radius thereof is set to radius R1, as shownin FIG. 7.

An amount of deviation (X0-X1) in the horizontal direction of the center(center of the circle) of a mask of the left- and right-eye images istypically constant, and thus it may be possible to measure the amount ofdeviation in advance. In addition, when the amount of deviation ischanged, the distance between the centers of a mask of the left- andright-eye images may be measured dynamically.

In the image processing device 200, the endoscopic image for the lefteye is acquired by the image acquisition unit 211-1 and is supplied tothe mask detection unit 221-1. The mask detection unit 221-1 outputsinformation regarding the mask shape in the endoscopic image for theleft eye to the mask detection result buffer 222-1 and the inter-framemask detection result comparison unit 223-1.

The inter-frame mask detection result comparison unit 223-1 isconfigured to include a determination unit and an output unit (both notshown), in the same way to the inter-frame mask detection resultcomparison unit 123 according to the first embodiment. The inter-framemask detection result comparison unit 223-1 corrects a mask shape inconsideration of consistency in the detection results between framesusing the supplied information regarding the mask shape, and outputsinformation regarding the corrected mask shape to the inter-stereo maskdetection result comparison and correction unit 213.

As described later, the inter-frame mask detection result comparisonunit 223-1 generates a reliability parameter and outputs the generatedreliability parameter to the inter-stereo mask detection resultcomparison and correction unit 213.

The inter-frame mask detection result comparison unit 223-1 performs acomparison with the mask shape after correction at time t−1 (X′_(t-1),Y′_(t-1), R′_(t-1)) in the processing by the inter-frame mask detectionresult comparison unit.

The first embodiment is different from the second embodiment in that themask shape after correction at time t−1 in the first embodiment is theresult outputted from the inter-frame mask detection result comparisonunit 123 at time t−1, but the mask shape after correction at time t−1 inthe second embodiment is the result outputted from the inter-stereo maskdetection result comparison and correction unit 213.

Similarly, in the image processing device 200, an endoscopic image forthe left eye is acquired by the image acquisition unit 211-2 and issupplied to the mask detection unit 221-2. The mask detection unit 221-2outputs information regarding the mask shape in the endoscopic image forthe right eye to the mask detection result buffer 222-2 and theinter-frame mask detection result comparison unit 223-2.

The inter-frame mask detection result comparison unit 223-2 corrects amask shape in consideration of consistency in the detection resultsbetween frames using the supplied information regarding the mask shapeand outputs information regarding the corrected mask shape to theinter-stereo mask detection result comparison and correction unit 213,in the same way to the inter-frame mask detection result comparison unit123 in the first embodiment.

As described later, the inter-frame mask detection result comparisonunit 223-2 generates a reliability parameter and outputs the generatedreliability parameter to the inter-stereo mask detection resultcomparison and correction unit 213.

The inter-stereo mask detection result comparison and correction unit213 compares the mask detection result in the left endoscopic image andthe mask detection result in the right endoscopic image, and correctsthe result outputted from the inter-frame mask detection resultcomparison unit 223 depending on the magnitude of the absolute value ofthe difference between them.

<Operation by Image Processing Device According to Second Embodiment>

Referring to the flowchart of FIG. 8, the operation performed by theimage processing device 200 shown in FIG. 6 is described. Specifically,the operation performed by the inter-frame mask detection resultcomparison unit 223 of the mask detection and correction unit 212 ismainly described.

The operation performed by the inter-frame mask detection resultcomparison unit 223 is performed in a substantially similar way to theoperation performed by the inter-frame mask detection result comparisonunit 123 according to the first embodiment. The difference between themis that a reliability parameter is calculated and outputted, and thusthe description is made about the difference.

Processing in steps S201 to S204 is similar to that in steps S101 toS104 of the flowchart shown in FIG. 3, processing in steps S206 and S207is similar to that in steps S105 and S106, and processing in steps S210to S218 is similar to that in steps S108 to S116. Thus, the repeateddescription will be omitted.

In step S204, if it is determined that the inequalities (4) to (6) areall established, then the process proceeds to step S205. In step S205,the reliability parameter (Conf) is set to one. The following will bedescribed on the assumption that the reliability parameter is set to oneif the mask shape to be outputted (parameter regarding a mask) is morelikely to be correct, but the reliability parameter is set to zero ifthe mask shape to be outputted is less likely to be correct.

In other words, the reliability parameter is set to one if it isdetermined that the mask shape after correction outputted from theinter-frame mask detection result comparison unit 213 is substantiallyconsistent with the mask shape detected by the mask detection unit 221(the inequalities (4) to (6) are all established) and the mask shape isnot changed. The reliability parameter is set to zero if it isdetermined that the mask shape is likely to be changed.

In this regard, the following description will be given on theassumption that the reliability parameter is set to zero or one, but thecondition in which the reliability parameter may be set to zero or onemay be reversed. In addition, the reliability parameter may be set to avalue other than zero or one. For example, the reliability parameter maybe set to a value corresponding to the possibility that the mask shapeis changed.

In step S208, the reliability parameter that is set in this way isoutputted to the inter-stereo mask detection result comparison andcorrection unit 213 together with the mask detection result aftercorrection at time t.

On the other hand, in step S204, if a condition that the inequalities(4) to (6) are all established is determined not to be satisfied, thenthe process proceeds to step S209. In step S209, the reliabilityparameter (Conf) is set to zero. In this case, the mask shape is likelyto be changed, and thus the reliability parameter is set to zero.

After step S209, processing in step S210 and the subsequent steps aresubstantially similar to that of the first embodiment, thus adescription thereof will be omitted.

As described above, the inter-frame mask detection result comparisonunit 223-1 processes an image for the left eye and supplies the maskdetection result after correction and the reliability parameter to theinter-stereo mask detection result comparison and correction unit 213.Similarly, as described above, the inter-frame mask detection resultcomparison unit 223-2 processes an image for the right eye and suppliesthe mask detection result after correction and the reliability parameterto the inter-stereo mask detection result comparison and correction unit213.

Referring now to the flowchart of FIG. 9, the operation performed by theinter-stereo mask detection result comparison and correction unit 213 isdescribed.

In step S301, a reliability parameter Conf0 and information regardingthe mask shape after correction (X0′_(t), Y0′_(t), R0′_(t)) in theendoscopic image for the left eye outputted from the inter-frame maskdetection result comparison unit 223-1 are obtained.

In step S302, a reliability parameter Conf1 and information regardingthe mask shape after correction (X1′_(t), Y1′_(t), R1′_(t)) in theendoscopic image for the left eye are obtained.

In step S303, the absolute value of the difference between the acquiredrespective parameters, ADS_(X), ADS_(Y), and ADS_(R) are calculatedbased on the following Equations (19) to (21).

In the following Equations (19) to (21), ADS_(X) represents the absolutevalue of the difference between x-coordinates of the center point of themask shape for the left eye and the right eye after correction, ADS_(Y)represents the absolute value of the difference between y-coordinates ofthe center point of the mask shape for the left eye and the right eyeafter correction, and ADS_(R) represents the absolute value of thedifference between the radii R of the mask shape for the left eye andthe right eye after correction.

ADS _(X) =|X0′_(t) −X1′_(t)|  (19)

ADS _(Y) =Y0′_(t) −Y1′_(t)|  (20)

ADS _(R) =|R0′_(t) −R1′_(t)|  (21)

In step S304, it is determined whether the following Equations, that is,inequalities (22) to (25) are all established. In the inequalities (22)to (25), the thresholds TH_MIN_ADS_(X) and TH_MAX_ADS_(X) are thresholdvalues for the absolute value of x-coordinate of the center point. Thethreshold TH_ADS_(Y) is a threshold for the absolute value ofy-coordinate of the center point, and the threshold TH_ADS_(R) is athreshold for the absolute value of the radius R.

Threshold,TH_MIN_(—) ADS _(X)<Absolute value of x-coordinate of centerpoint,ADS _(X)  (22)

Absolute value of x-coordinate of center point,ADS_(X)<Threshold,TH_MAX_(—) ADS _(X)  (23)

Absolute value of y-coordinate of center point,ADS _(Y)<Threshold,TH_(—) ADS _(Y)  (24)

Absolute value of radius R,ADS _(R)<Threshold,TH _(—) ADS _(R)  (25)

In step S304, if it is determined that Equations (22) to (25) are allestablished, the process proceeds to step S305. In other words, thiscase occurs when the absolute value of the difference ADS_(X) is greaterthan or equal to the threshold TH_MIN_ADS_(X) and less than thethreshold TH_MAX_ADS_(X) and when the absolute values of the differenceADS_(Y) and ADS_(R) are less than the thresholds TH_ADS_(Y) andTH_ADS_(R), respectively.

In such cases, in step S305, the results outputted from the inter-framemask detection result comparison unit 223-1 and the inter-frame maskdetection result comparison unit 223-2 are outputted, without anymodification, from the inter-stereo mask detection result comparison andcorrection unit 213.

In step S304, if a condition that Equations (22) to (25) are allestablished is determined not to be satisfied, the process proceeds tostep S306. In step S306, the reliability parameter Conf0 from theinter-frame mask detection result comparison unit 223-1 and thereliability parameter Conf1 from the inter-frame mask detection resultcomparison unit 223-2 are compared with each other.

In step S306, it is determined whether the reliability parameter of theleft-eye image Conf0 is set to one and whether the reliability parameterof the right-eye image Conf1 is set to zero. If such a condition issatisfied, it can be determined that the result in the left-eye imageoutputted from the inter-frame mask detection result comparison unit223-1 is more reliable than the result in the right-eye image outputtedfrom the inter-frame mask detection result comparison unit 223-2.

In step S306, if it is determined that the reliability parameter of theleft-eye image Conf0 is set to one and the reliability parameter of theright-eye image Conf1 is set to zero, the process proceeds to step S307.

In step S307, as a mask shape after correction for the left-eye imagefrom the inter-stereo mask detection result comparison and correctionunit 213, the result (X0′_(t), Y0′_(t), R0′_(t)) outputted from theinter-frame mask detection result comparison unit 223-1 is outputtedwithout any modification. In this case, the mask shape in the left-eyeimage from the inter-frame mask detection result comparison unit 223-1is used without any modification because it is more likely to becorrect.

On the other hand, parameters of the mask shape after correction in theright-eye image are calculated based on the following Equations (26) to(28).

X1′_(t) =X0′_(t) +D ₁₀  (26)

Y1′_(t) =Y0′_(t)  (27)

R1′_(t) =R0′_(t)  (28)

In Equation (26), D₁₀ represents the distance between the centers of themask for the left- and right-eye images, as shown in the followingEquation (29).

D ₁₀ =X1−X0  (29)

In Equation (29), X0 represents x-coordinate of the center point of themask shape for the left-eye image at a given time, and X1 representsx-coordinate of the center point of the mask shape for the right-eyeimage at a given time. For example, X0 and X1 may be previouslydetermined, or they may be calculated when the reliability parameter forthe left-eye image Conf1 of the inter-frame mask detection resultcomparison unit 223-1 and the reliability parameter for the right-eyeimage Conf1 of the inter-frame mask detection result comparison unit223-2 are all set to one.

The parameters of the mask shape after correction in the right-eye imagecalculated in step S307 and the result outputted from the inter-framemask detection result comparison unit 223-1 that is set as the maskshape after correction for the left-eye image are outputted to thesubsequent processing unit, in step S305.

On the other hand, in step S306, if a condition that the reliabilityparameter of the left-eye image Conf0 is set to one and the reliabilityparameter of the right-eye image Conf1 is set to zero is determined notto be satisfied, the process proceeds to step S308.

In step S308, it is determined whether the reliability parameter of theleft-eye image Conf0 is set to zero and the reliability parameter of theright-eye image Conf1 is set to one. If such a condition is satisfied,it can be determined that the result in the right-eye image outputtedfrom the inter-frame mask detection result comparison unit 223-2 is morereliable than the result in the left-eye image outputted from theinter-frame mask detection result comparison unit 223-1.

In step S308, if it is determined that he reliability parameter of theleft-eye image Conf0 is set to zero and the reliability parameter of theright-eye image Cont1 is set to one, the process proceeds to step S309.

In step S309, as a mask shape after correction for the right-eye imagefrom the inter-stereo mask detection result comparison and correctionunit 213, the result (X1′_(t), Y1′_(t), R1′_(t)) outputted from theinter-frame mask detection result comparison unit 223-2 is outputtedwithout any modification.

On the other hand, parameters of the mask shape after correction in theleft-eye image are calculated based on the following Equations (30) to(32).

X0′_(t) =X1′_(t) +D ₀₁  (30)

Y0′_(t) =Y1′_(t)  (31)

R0′_(t) =R1′_(t)  (32)

In Equation (30), D₀₁ represents the distance between the centers of themask for the left- and right-eye images, as shown in the followingEquation (33).

D ₀₁ =X0−X1  (33)

In Equation (33), X0 represents x-coordinate of the center point of themask shape for the left-eye image at a given time, and X1 representsx-coordinate of the center point of the mask shape for the right-eyeimage at a given time. For example, X0 and X1 may be previouslydetermined, or they may be calculated when the reliability parameter forthe left-eye image Conf0 of the inter-frame mask detection resultcomparison unit 223-1 and the reliability parameter for the right-eyeimage Conf1 of the inter-frame mask detection result comparison unit223-2 are all set to one.

The parameters of the mask shape after correction in the left-eye imagecalculated in step S309 and the result outputted from the inter-framemask detection result comparison unit 223-2 that is set as the maskshape after correction for the right-eye image are outputted to thesubsequent processing unit, in step S305.

On the other hand, in step S308, if a condition that the reliabilityparameter of the left-eye image Conf0 is set to zero and the reliabilityparameter of the right-eye image Conf1 is set to one is determined notto be satisfied, the process proceeds to step S310.

The process proceeds to step S310 when the reliability parameter of themask detection result in the left-eye image Conf0 is set to zero and thereliability parameter of the mask detection result in the right-eyeimage Conf1 is set to zero. In such a case, it can be determined thatthe mask detection result in the left-eye image and the mask detectionresult in the right-eye image have low reliability.

Alternatively, the process proceeds to step S310 when the reliabilityparameter of the mask detection result in the left-eye image Conf0 isset to one and the reliability parameter of the mask detection result inthe right-eye image Conf1 is set to one. In such a case, it can bedetermined that the mask detection result in the left-eye image and themask detection result in the right-eye image have high reliability.

When both of the mask detection result in the left-eye image and themask detection result in the right-eye image have high reliability orlow reliability, it is difficult to determine which of the maskdetection results in the left-eye image and in the right-eye image ismore reliable and which of them is proper to be used for correction.

In such a case, in step S310, based on the mask detection result in theimage that is previously set among ones outputted from the inter-framemask detection result comparison unit 223, the mask shape of the otherimage is corrected.

For example, if a mask detection result in the left-eye image is set tobe processed, the inter-stereo mask detection result comparison andcorrection unit 213 outputs, as the mask shape after correction for theleft-eye image, the result (X0′_(t), Y0′_(t), R0′_(t)) outputted fromthe inter-frame mask detection result comparison unit 223-1 without anymodification, and outputs, as the mask shape after correction for theright-eye image, the parameter calculated based on Equations (26) to(28).

Which of the left-eye image and the right-eye image is to be processedmay be determined previously, or processing of checking a change in thereliability parameters Conf0 and Conf1 and switching dynamically intoone having higher reliability may be performed.

The parameter that is set in this way is outputted to the subsequentprocessing unit as the mask shape information (X0′_(t), Y0′_(t),R0′_(t)) of the left-eye image and mask shape information (X1′_(t),Y1′_(t), R1′_(t)) of the right-eye image after correction at time t instep S305 as described above, and then the processing in theinter-stereo mask detection result comparison and correction unit 213 isended.

Furthermore, the mask shape information of the left-eye image afterfinal correction at time t (X0′_(t), Y0′_(t), R0′_(t)) is stored in themask detection result buffer 222-1, and the mask shape information ofthe right-eye image after final correction at time t (X1′_(t), Y1′_(t),R1′_(t)) is stored in the mask detection result buffer 222-2, and eachinformation is used for the next frame.

Such processing will be described in detail with reference to FIG. 10.

In FIG. 10, the horizontal axis t represents time and, images (frames)of the figures are images obtained in the past in the order of timetoward the left side. In addition, the upper part in the figurerepresents a left-eye image to be processed or has been processed by themask detection and correction unit 212-1 (see FIG. 6), and the lowerpart in the figure represents a right-eye image to be processed or hasbeen processed by the mask detection and correction unit 212-2 (see FIG.6). In FIG. 10, the mask region 252 is shown without being blackened.

In the figure, the thick solid-line circles 271 and 281 represent theresult (mask detection result before correction) outputted from the maskdetection unit 221 (see FIG. 6). The thin solid-line circles 272 and 282in the figure represent the result (mask detection result aftercorrection between frames) outputted from the inter-frame mask detectionresult comparison unit 223. The dotted-line circles 273 and 283 in thefigure represent the result (mask detection result after finalcorrection) outputted from the inter-stereo mask detection resultcomparison and correction unit 213.

A cloud-shaped mark in the figure represents a case in which the imageis in a deteriorated condition for some reasons such as adhesion of dustor dirt on the lens included in an endoscopic device and then normalmask detection is not allowed to be performed. In the example shown inFIG. 10, such a situation is occurred in the frames at time T43 and timeT44 in the left-eye image, and is occurred in the frames at time T41 toT47 in the right-eye image.

The following description will be given on the assumption that thethreshold TH_CNT of the count value Cnt of the inter-frame maskdetection result comparison unit 223 is set to five.

At time T41, in the right-eye image, the result (circle 281) outputtedform the mask detection unit 221-2 is erroneous, but it is corrected bythe inter-frame mask detection result comparison unit 223-2, whichperforms the right-eye image, using the result (circle 283) outputtedfrom the inter-stereo mask detection result comparison and correctionunit 213 at time T40. Thus, the absolute value of the difference betweenframes in the left- and right-eye images in the inter-stereo maskdetection result comparison and correction unit 213 is within the rangeof threshold values.

At time T42, processing similar to that performed at time T41 isperformed.

At time T43, in both the left-eye and right-eye images, the results(circles 271 and 281) outputted form the mask detection unit 221 areerroneous, but the results are corrected (circles 272 and 282) by theinter-frame mask detection result comparison units 223-1 and 223-2,respectively.

At time T44, processing similar to that performed at time T43 isperformed.

At time T45, in the right-eye image, an erroneous mask detection of themask detection unit 221-2 is continued, and thus the count value Cnt ofthe inter-frame mask detection result comparison unit 223-2 satisfiesthe threshold TH_CNT (=5). The result (circle 282) outputted from theinter-frame mask detection result comparison unit 223-2 is corrected tobe the same as the result (circle 281) outputted from the mask detectionunit 221-2.

At time T45, the reliability parameter Conf1 in the inter-frame maskdetection result comparison unit 223-2 in the right-eye image is set tozero.

On the other hands, the detection result in the left-eye image at timeT45 is correct. In the inter-frame mask detection result comparison unit223-1, the absolute value of the difference with the result (circle 273)outputted from the inter-stereo mask detection result comparison andcorrection unit 213 at time T44 is within the threshold, and thus thereliability parameter Conf1 is set to one.

In the inter-stereo mask detection result comparison and correction unit213, the absolute value of the difference of the results outputted fromthe respective inter-frame mask detection result comparison units 223for the left-eye image and the right-eye image falls outside thethreshold, and the mask shape for the right-eye image is corrected basedon the result (circle 272) outputted from the inter-frame mask detectionresult comparison unit 223-1 for the left-eye image by the comparison ofthe reliability (Conf0=1 and Conf1=0). This correction is performedbased on Equations (26) to (28).

In this way, both of the inter-frame mask detection result comparisonunit 223 and the inter-stereo mask detection result comparison andcorrection unit 213 perform correction, and thus more stable detectionof a mask is possible.

<When Operation is Started>

The mask detection and correction unit 112 of the image processingdevice 100 according to the first embodiment or the mask detection andcorrection unit 212 of the image processing device 200 according to thesecond embodiment may be configured to perform their respectiveoperations in a normal condition or when a predetermine event occurs.

In the examples described above, the case in which the operation isperformed in a normal condition, that is, for every frame has beendescribed as an example. The following description will be given byexemplifying the image processing device 100 according to the firstembodiment. When the operation is performed in a normal condition, themask shape detected by the mask detection unit 121 is stored in the maskdetection result buffer 122 for every frame, is corrected by theinter-frame mask detection result comparison unit 123, and is outputtedas the mask shape after correction.

If the operation is performed when a predetermined event occurs, theparameter regarding the mask shape detected by the mask detection unit121 for every frame is stored in the mask detection result buffer 122,and thus even when an event occurs at any timing, it is possible to dealwith the event.

When a predetermined event occurs, correction by the inter-frame maskdetection result comparison unit 123 is started using the parameterstored in the mask detection result buffer 122. Such configuration maybe applied to the second embodiment.

The predetermined event includes an event when it is determined that anerror occurs. Whether an error occurs can be determined, for example,using an average luminance level of the in-vivo region (effective region151) in the image from the endoscopic device.

As an example, when the average luminance level is low, it is difficultto define the boundary between the mask region 152 and the in-vivoregion (effective region 151), and thus detection of a mask is difficultto perform. In such a case, in other words, when the average luminancelevel is less than a predetermined threshold, the correction describedabove may be started on the assumption that an error occurs (occurrenceof event).

Moreover, as another example of the predetermined event, thedetermination as to whether a rigid lens is inserted into a trocar ispossible, and the event can be regarded as when the rigid lens isinserted into the trocar or the inserted rigid lens is released. In sucha configuration, for example, the determination as to whether the rigidlens is inserted into the trocar is possible using a sensor attached tothe trocar.

A mask shape obtained immediately before the rigid lens is removed maybe stored, and the stored mask shape may be used until the rigid lens isinserted again.

Furthermore, as another example of the predetermined event, the maskshape may be corrected by considering, as an event, a mechanism forallowing the user to instruct correction of a mask, for example, forproviding a button for such instruction and operating the button. Forexample, it is possible to correct the mask shape by considering, as anevent, a mechanism for providing a foot switch and operating the footswitch.

It is possible to configure so that the mask shape may be correctedusing such an external input.

<Recording Medium>

The series of processes described above can be executed by hardware butcan also be executed by software. When the series of processes isexecuted by software, a program that constructs such software isinstalled into a computer. Here, the expression “computer” includes acomputer in which dedicated hardware is incorporated and ageneral-purpose personal computer or the like that is capable ofexecuting various functions when various programs are installed.

FIG. 11 is a block diagram illustrating a hardware configuration exampleof a computer for causing the above-described series of processes to beexecuted using a program. In the computer, a central processing unit(CPU) 1101, a read only memory (ROM) 1102, and a random access memory(RAM) 1103 are interconnected via a bus 1104. The bus 1104 is connectedto an input/output interface 1105. The input/output interface 1105 isconnected to an input unit 1106, an output unit 1107, a storage unit1108, a communication unit 1109, and a drive 1110.

The input unit 1106 includes a keyboard, a mouse, a microphone, andother like devices. The output unit 1107 includes a display, a speaker,and other like devices. The storage unit 1108 includes a hard disk, anon-volatile memory, and other like devices. The communication unit 1109includes a network interface and other like devices. The drive 1110drives a removable medium 1111 such as a magnetic disk, an optical disk,a magneto-optical disk, a semiconductor memory or the like.

In the computer configured as described above, as one example the CPU1101 loads a program stored in the storage unit 1108 via theinput/output interface 1105 and the bus 1104 into the RAM 1103 andexecutes the program to carry out the series of processes describedearlier.

Programs to be executed by the computer (CPU 1101) are provided beingrecorded in the removable medium 1111 in the form of a packaged mediumor the like. The programs may be provided via a wired or wirelesstransmission medium, such as a local area network, the Internet, ordigital satellite broadcasting.

In the computer, by inserting the removable medium 1111 into the drive1110, the program can be installed in the storage unit 1108 via theinput/output interface 1105. Further, the communication unit 1109 canreceive the program via a wired or wireless transmission medium and caninstall it in the storage unit 1108. Moreover, the program can beinstalled in advance in the ROM 1102 or the storage unit 1108.

It should be noted that the program executed by a computer may be aprogram that is processed in time series according to the sequencedescribed herein or a program that is processed in parallel or atnecessary timing such as upon calling.

Note that the term “system” used herein refers to an entireconfiguration composed of a plurality of devices.

Note that the advantages described herein are to be consideredillustrative or exemplary rather than restrictive, and other advantagesthat will be understood from the present technology may be achievable.

An embodiment of the technology is not limited to the embodimentsdescribed above, and various changes and modifications may be madewithout departing from the scope of the technology.

Additionally, the present technology may also be configured as below.

(1) An image processing device including:

a detection unit configured to detect a mask from an acquired image;

a determination unit configured to determine whether there is a changein the mask detected by the detection unit; and

an output unit configured to output a parameter when the determinationunit determines that there is a change in the mask, the parameter beingrelated to the mask detected by the detection unit before it isdetermined that there is a change in the mask.

(2) The image processing device according to (1),

wherein the determination unit determines whether there is a change inthe mask based on a temporal change in a parameter of the mask detectedby the detection unit.

(3) The image processing device according to (2),

wherein the determination unit determines that there is no change in themask detected by the detection unit when a difference between aparameter of a first mask detected by the detection unit and a parameterof a second mask detected by the detection unit after the first mask isdetected is less than a predetermined threshold, and

wherein the output unit outputs a parameter of a third mask when thedetermination unit determines that there is no change in the mask, theparameter of the third mask being calculated from the parameter of thefirst mask and the parameter of the second mask.

(4) The image processing device according to (2),

wherein the determination unit determines that there is a change in themask detected by the detection unit when a difference between aparameter of a first mask detected by the detection unit and a parameterof a second mask detected by the detection unit after the first mask isdetected is greater than or equal to a predetermined threshold, and

wherein the output unit outputs the parameter of the first mask when thedetermination unit determines that there is a change in the mask.

(5) The image processing device according to (4),

wherein the output unit outputs the parameter of the second mask insteadof the parameter of the first mask when the determination unitdetermines that there is a change in the mask by a predetermined numberof times.

(6) The image processing device according to (4),

wherein the output unit outputs a parameter of a third mask calculatedfrom the parameter of the first mask and the parameter of the secondmask when the determination unit determines that there is a change inthe mask by a predetermined number of times.

(7) The image processing device according to any one of (1) to (6),

wherein the acquired image is an image captured by an endoscope.

(8) The image processing device according to (1), further including:

a correction unit configured to correct a parameter of the mask using areliability parameter representing reliability of a parameter of themask.

(9) The image processing device according to (1),

wherein the detection unit detects a first mask and a second mask fromtwo respective acquired images, and

wherein the image processing device further includes a correction unitconfigured to determine whether at least one of a parameter of the firstmask and a parameter of the second mask is to be corrected, based on theparameter of the first mask and the parameter of the second mask.

(10) The image processing device according to (9),

wherein the correction unit corrects one of the parameter of the firstmask and the parameter of the second mask using the other parameter thatis set previously, when a difference between the parameter of the firstmask and the parameter of the second mask is outside a predeterminedrange.

(11) The image processing device according to (9),

wherein the correction unit performs correction using a reliabilityparameter representing reliability of each of the parameter of the firstmask and the parameter of the second mask when a difference between theparameter of the first mask and the parameter of the second mask isoutside a predetermined range.

(12) The image processing device according to any one of (1) to (11),

wherein the determination unit determines whether there is a change inthe mask based on a luminance level of the image.

(13) The image processing device according to (12),

wherein the determination unit determines that there is a change in themask when an average luminance level of the image is less than apredetermined threshold.

(14) The image processing device according to any one of(1) to (13),

wherein the determination unit determines whether there is a change inthe mask based on whether an endoscope is inserted into a trocar.

(15) The image processing device according to (14),

wherein the determination unit determines that there is a change in themask when it is determined that an endoscope is not inserted into atrocar.

(16) The image processing device according to any one of (1) to (15),

wherein the determination unit determines whether there is a change inthe mask based on an external input.

(17) The image processing device according to (16),

wherein the external input is provided using a foot switch.

(18) An image processing method including:

detecting a mask from an acquired image;

determining whether there is a change in the detected mask; and

outputting a parameter when it is determined that there is a change inthe mask, the parameter being related to the mask detected before it isdetermined that there is a change in the mask.

What is claimed is:
 1. An image processing device comprising: adetection unit configured to detect a mask from an acquired image; adetermination unit configured to determine whether there is a change inthe mask detected by the detection unit; and an output unit configuredto output a parameter when the determination unit determines that thereis a change in the mask, the parameter being related to the maskdetected by the detection unit before it is determined that there is achange in the mask.
 2. The image processing device according to claim 1,wherein the determination unit determines whether there is a change inthe mask based on a temporal change in a parameter of the mask detectedby the detection unit.
 3. The image processing device according to claim2, wherein the determination unit determines that there is no change inthe mask detected by the detection unit when a difference between aparameter of a first mask detected by the detection unit and a parameterof a second mask detected by the detection unit after the first mask isdetected is less than a predetermined threshold, and wherein the outputunit outputs a parameter of a third mask when the determination unitdetermines that there is no change in the mask, the parameter of thethird mask being calculated from the parameter of the first mask and theparameter of the second mask.
 4. The image processing device accordingto claim 2, wherein the determination unit determines that there is achange in the mask detected by the detection unit when a differencebetween a parameter of a first mask detected by the detection unit and aparameter of a second mask detected by the detection unit after thefirst mask is detected is greater than or equal to a predeterminedthreshold, and wherein the output unit outputs the parameter of thefirst mask when the determination unit determines that there is a changein the mask.
 5. The image processing device according to claim 4,wherein the output unit outputs the parameter of the second mask insteadof the parameter of the first mask when the determination unitdetermines that there is a change in the mask by a predetermined numberof times.
 6. The image processing device according to claim 4, whereinthe output unit outputs a parameter of a third mask calculated from theparameter of the first mask and the parameter of the second mask whenthe determination unit determines that there is a change in the mask bya predetermined number of times.
 7. The image processing deviceaccording to claim 1, wherein the acquired image is an image captured byan endoscope.
 8. The image processing device according to claim 1,further comprising: a correction unit configured to correct a parameterof the mask using a reliability parameter representing reliability of aparameter of the mask.
 9. The image processing device according to claim1, wherein the detection unit detects a first mask and a second maskfrom two respective acquired images, and wherein the image processingdevice further includes a correction unit configured to determinewhether at least one of a parameter of the first mask and a parameter ofthe second mask is to be corrected, based on the parameter of the firstmask and the parameter of the second mask.
 10. The image processingdevice according to claim 9, wherein the correction unit corrects one ofthe parameter of the first mask and the parameter of the second maskusing the other parameter that is set previously, when a differencebetween the parameter of the first mask and the parameter of the secondmask is outside a predetermined range.
 11. The image processing deviceaccording to claim 9, wherein the correction unit performs correctionusing a reliability parameter representing reliability of each of theparameter of the first mask and the parameter of the second mask when adifference between the parameter of the first mask and the parameter ofthe second mask is outside a predetermined range.
 12. The imageprocessing device according to claim 1, wherein the determination unitdetermines whether there is a change in the mask based on a luminancelevel of the image.
 13. The image processing device according to claim12, wherein the determination unit determines that there is a change inthe mask when an average luminance level of the image is less than apredetermined threshold.
 14. The image processing device according toclaim 1, wherein the determination unit determines whether there is achange in the mask based on whether an endoscope is inserted into atrocar.
 15. The image processing device according to claim 14, whereinthe determination unit determines that there is a change in the maskwhen it is determined that an endoscope is not inserted into a trocar.16. The image processing device according to claim 1, wherein thedetermination unit determines whether there is a change in the maskbased on an external input.
 17. The image processing device according toclaim 16, wherein the external input is provided using a foot switch.18. An image processing method comprising: detecting a mask from anacquired image; determining whether there is a change in the detectedmask; and outputting a parameter when it is determined that there is achange in the mask, the parameter being related to the mask detectedbefore it is determined that there is a change in the mask.